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TASK FACTOR BENCHMARK SCALES 
FOR TRAINING PRIORITY ANALYSIS: 
OVERVIEW AND DEVELOPMENTAL PHASE 
FOR ADMINISTRATIVE/GENERAL APTITUDE AREA 



I. INTRODUCTION 

The Otxupational and Manpower Research Division of the Air Force Human Resources Laboratoiy 
(AFHRL) is engaged ?n ongoing research into an advanced methodology for determining task training 
priorities. One etement of this research is the development of bendimaik scales for measuiing task factors 
that contribute to training priority decisions. To put the bendimark scales prcject into context, this report 
will comment briefly on the over^ training priority research effort before detailing the procedure adopted 
to develop the scales. Finally, it will report the results of the Jnitad data gathering and tentative scale 
drafting phase for the sf;ales for tho^e spedalties with administrative or general aptitude requirements. 



IL BACKGROUND 

Tae basic co?.cept of the present tusk training priority research was conceived and reported by 
Christal (1970). 'Ihrce papers lead to the 17th Annual Conference of the Military Testing Association 
(MTA) document achievements since then. The first of these papers (Christal & Weissmuller, 1975), aow 
also available as a technical report (Chnstal & Weissmuller, 1976), describes eig::t new programs recently 
introduced into the Comprehensive Occupational Data Analysis Programs (CODAF) system which enable 
investigatoi^ to manipulate and analyze task factor data. It indudes an example wldch demonstrates how 
the programs can be used in developing and applying an equation to determine task training priorities. The 
second MTA paper (Mead, 1975) reports the ii::tults of a training priority study; and the third (Goody & 
Watson, 1975) introduces the benchmark scales and is encompassed in this report. Mead's paper points out 
how AFHRL has identified four task factors which appear to be the best determinants of a task's priority 
for formal training. They are: 

1. Percentage of first*term airmen performing the task. 

2. Probable consequences of inadequate performance of the task. 

3. The delay that can be tolerated between the time the need for a task becomes evident and the 
time when actual performance must begin. 

4. The amount of time required to learn to perform the task adequately— the task difflculty. 

Mead also indicates how the CODAP programs described in the first of thsse three papers (Christal & 
Weissmuller, 1975), can be used to arrange task factor data in a format most suitable for use by training 
managers in making better training dedsions. The feasibility of using regression equations to compute a 
training priority index from task factor data, postulated by Christal, has been confirmed. The new CODAP 
programs can then list aD the tasks in a specialty ordered from high to low on this index, presenting 
pertinent task factor data beside ich task. Hie tasks neat th.? • r this list are those on which training is 
essential, and those near the bottcwn are the ones with low priority for training. Li between these two 
extremes are all the **could be trained" tasks listed in order of their priority for universal training. 

Thiii methodology would have various applications. It could be used for the initial design of a training 
program, the validation of an existing program, or for the redesign of an existing one given new constraints 
such as change in course length. At all times it must be remembered that this methodology is an advanced 
aid to course design, always subject to human override wliere such can be justified* Perhaps the biggest 
danger to this project is the pot^tial fo' bHnd apldication of the mathematical model without due 
consideration for the exceptional case. However, provided training management is aware of this, the model 
offers vast potential as an aid to optimum training dedsions. 

The major limitation in the methoddogy at present is the method used for measuring the task 
factors, apart from the percent-performing factor which is an absolute value. The factors are measured by 

8 

5 

ERIC 



rating tasks relative to other tasks in a specialty. Specialists consider each task in turn, and then allocate it a 
rating relative to all other tasks in the specialty. The arithmetic mean of all ratings for a task is used as the 
measure of that factor fo» that task. The problem is that the numerical value so ascribed to a task depends 
on the nature of the other tasks in the specialty. To illustrate, consider two tasks with equal degrees of 
**Conscquences of Inadequate Performance," one a Recruiter task and the other an Operating Room 
Specialist task. Operating Room Specialist tasks, on the whole, tend to be more consequential than 
Recruiter tasks. In our example, therefore, although the consequences of failing to adequately perform the 
two tasks are equal, the Recruiter task will be ascribed a higher numerical value because it is being 
compared to relatively less consequential tasks. Such ratings can be used to determine an optimum 
regression equation within a specialty. However, a new regression equation must be computed for each 
specialty. The possibility exists that, provided the task factors can be measured on common scales, a limited 
number of regression equations can be computed, each applying across a number of specialties. 

In order to examine this possibility a series of "benchmaric scales'' are being developed which should 
permit measurement of task factors against common frames of reference for various specialties. To illustrate 
the concept of a benchmark scale. Appendix A contains the tentative benchmark scales for the three factors 
for those specialties with an Administrative or General aptitude requirement. Each comprises 27 tasks, 
divided into nine subgroups of three tasks, each subgroup' representing a level on the relevant factor. In 
operational use, specialist raters will rale each of the tasks in their specialty against the 27 tasks of the 
benchmark scale rather than agaiiist the other lasks of their own specialty. For each task the rater will 
decide which ^roup of three tasks in the scale has about the same degree of the factor under consideration, 
the level assL iated with that group of three being his ratmg for the task under consideration. 

In all, nine scales are being developed, one for each of three factors for each of three aptitude areas. 
The three factors are Consequences of Inadequate Performance, Task Delay Tolerance, and Task Difficulty, 
definitions of each being included in Appendix A. The three groups of specialties are defined on the basis of 
aptitude requirements: the first contains specialties with an Administrative or General requirement, and the 
other two gioups are those with an Electronic or Mechanical aptitude requirement, respectively. 



III. PROCEDURE USED IN THE DEVELOPMENT OF 
TRAINING FACTOR BENCHMARK SCALES 

The development of a benchmark scale is a two-phase process. Initially, data is gathered and analyzed 
for a reasonably large inventory of tasks; this data is then used to select the 27 tasks for the tentative 
benchmark scale. In the second phase the tentative scales are validated. 

The initial inventory contains tasks typically performed in a variety of specialties, but each must be 
of such a nature as to be underst^wd by experienced airmen from other specialties. As the present concem 
is for initial technical training, only joumeyman-level tasks are used. The inventory must contain tasks 
representing the fuU range of values on each of the factors for which it is used, and multi-facet tasks 
consisting of several divergent components are avoided. To aid identification of each task in the inventory, 
the task statement includes an Air Force specialty (AFS) that normally performs the task. (The benchmark 
scales in Appendix A iUustrate typical task statements.) 

For each factor, raters rate each inventory task relative to the other tasks in the inventory, using a 
9-point relative scale. The raters are selected randomly from the first-line supervision level (7-skill level) 
from specialties within the appropriate aptitude area, different raters being used for each factor. This is 
necessary to avoid spurious inter-action between factors; particularly as Task Delay Tolerance, with Ijevel 1 
the most demanding situation, is effectively an inverted scale relative to the other two. About 120 ratings 
are sought on each factor. Such large numbers are necessary to provide confidence in the stability of the 
means obtained by having raters rate tasks from other specialties. First-line supervisors were selected as 
raters in order to obtain the optimum blend of general experience and first-hand knowledge of 
joumeyman-level activities. 

The raw ratings are then refined and arranged into a format suitable for selecting the tasks for each 
benchmark scale. A standard computer program (Stacey, WeissmuUer, Barton, & Rogers, 1974) identifies 
the occasional rater who has not taken his task seriously, and he is eliminated from further processing. 
Some raters rate harshly, others are more lenient. To compensate, all the ratings for each rater are 
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standardized to a common mean of 5.0 and standard deviation of 1.0. Tlie mean and standard deviation of 
these adjusted ratings for a task are then taken respectively as the mos< appropriate measures of the relevant 
factor for that task and of rater agreement on tnat measure. For most of the tasks in the inventory, task 
difficulty ratings (relative wiihin specialties) have been collected during previous, routine occupational 
analysis. These ratings are extracted from the various Ov-^cupational survey reports for Comparison with the 
task difficulty ratings collected in this study. 

For each factor, the tasks are then ordered on the mean rating, this listing being used to select the 
nine groups of three tasks for the benchmark scale. All three tasks representing one level must lie near each 
other on this listing, but be relatively distant from the groups chosen to represent the levels on eitlier side. 
The magnitude of the difference in mean rating between successive groups n^ust be uniform, and preference 
is given to tasks with the lowest standard deviations-this indicating the best rai?r agreement. Tasks selected 
must provide maximum coverage of the various specialties, and they must be those most likely to be 
meaningful to airmen from other specialties. Similarities within groups and differences between groups 
must be apparent, as well as measured. In constructing the task difficulty scales, within specialty 
comparisons of these ratings with task difficulty ratings extracted from previous occupational analyses are 
included as a further consideration. Tasks for which there is a significant lack of agreement between these 
two measures are avoided in the selection of tasks for inclusion in the benchmark scales. 

The results of this initial phase of the developmental process is a list of 27 task statements for each 
factor. They arc presented in sets of tiiree, each set purporting to define one of nine graduated levels of that 
factor (see Appendix A). The validation phase tests whether raters can use these definitions of the levels to 
rate tasks on that factor. As already noted, each task statement in the initial inventory indudes the title of 
an Air Force specialty that is normally associated with the task. About ten of these specialties are selected, 
and ali the tasks associated with them in the initial inventory are extracted to form the validation phase 
inventory. This is a list of about a hundred task statements, the number associated with each of the 
represented specialties varying from as low as four to as high as twenty. For ea.:h factor, a set of raters is 
selected using the same criteria used for selection of raters in the initial phase. They are asked to give a task 
factor rating for each task listed in the validation phase inventory, using the nine levels as defined by the 
nine sets of three task statements in the benchmark scale. These may be thought of as "general" ratings to 
distinguish them from a second set of ratings also gathered during the validation phase. Th^^ second set, the 
"specialist" ratings, are obtained using the same rating tedininue as used for the "general" ratings; but the 
laters are selected only from the speciahies associated with task statements included in the validation phase 
inventory, and these raters rate only the tasks associated with their own specialty. For the tasks in the 
validation phase inventory there are now three sets of ratings for each factor; the "general'' ratings, the 
"specialist" ratings and, of course, the ratings on the corresponding tasks in the original inventory used in 
the initial phase. Statistical comparisons between these sets of ratings will pennit conclusions to be drawn 
about the effectiveness of the benchmark scale. 



IV. DEVELOPMEOT OF TENTATIVE SCALES FOR SPECIALTIES WITH 
ADMINISTRATIVE/GENERAL APTITUDE REQUIREMENTS 

For those specialties with Administrative or General aptitude requirements, the initial inventory has 
been administered and the data analyzed. The inventory comprised 438 tasks representing over 50 
specialties. Concern has been expressed as to whether supervisors can rate tasks from other specialties. As 
can be seen from the inter-rater agreement coefficients in Table 1, this is no longer a concem;such raters 
can rate tasks from other specialties with a high degree of agreement. 

A series of correlation coefficients were computed to examine the relationships between rating means 
and standard deviations. These correlation coefficients are presented in Tables 2, 3, and 4. Correlations 
were computed between the adjusted task means and standard deviations for each factor (Table 2). The 
existence of a significant correlation would indicate a relationship betwen rater agreement and mean rating. 
There is evidence that the mean and standard deviation for Task Delay Tolerance are positively related, 
suggesting that the raters showed more agreement on those tasks with low task delay tolerance. Table 3 
contains the conelations between the means for the three factors, and Table 4 between standard deviations. 
Significant between-factor correlations exist in all cases. The same tasks tend to be rated towards the more 
demanding end of each scale, and rater agreement depends to some extent on the task and/or its statement 
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Table 1. Inter-rater Agreement CoefHcients 



Factor 


N 






Consequences of Inadequate Performance 


116 


,51 


.99 


Task Delay Tolerance 


120 


44 


.99 


Task Difficulty 


117 


.48 


.99 


iaoi^ ^. v^orrelation Between Task Means 


and Standard Deviations 


Consequences of Inadequate Performance 






.01 


Task Delay Tolerance 






.22 


Task Difficulty 






.13 


Table 3. Correlation Between 


Task Means 






Factor 


A 


B 


c 


Consequences of Inadequate Performance 


1.00 






Task Delay Tolerance 


-.84 


1.00 




Task Difficulty 


.58 


-.38 


1.00 


Table 4. Coirelation Between Task Standard Deviations 


Factor 


A 


B 


c 


Consequences of Inadequate Pbrfomiance 


1.00 






Task Delay Tolerance 


.53 


1.00 




Task Difficulty 


.58 


.27 


1.00 
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regardless of the factor being rated. The hiph correlation between the mean ratings on the first two factors 
is understandable as delayed performaDce is o^'e form of inadequate performance, and the delay that can be 
tolerated is to some extent a function of the consequences of such delay. However, despite the strong 
velationship between these two factors, it is believed each has a sufficiently high unique component to 
contribute significantly to the training priority index model. 

As a first step in selecting the 2V lasks for the benchmark scale for each factor, the inventory tasks 
were divided into nine huerarchicai groups on the basis of mean adjusted ratings for the factor under 
consideration. The difference between the hughest and lowest mean rating in each of groups 2 through 8 
was half a standard deviation, the middle group straddling the mean and groups 1 and 9 being the lower and 
upper "tails.!' The set of three tasks representing each level was then selected from the corresponding 
group, observing the previously prescribed principles. To maximize the differencfis between adjacent sets 
while keeping them relatively uniform, the three tasks for level 5 were selected fiom as near the mean as 
possible (the middle of group 5), and those for the other levels were selected from the part of the 
corresponding group that is farthest from tn^ me.^. The tentative scales, developed and currently in the 
validation phase, appear in Appendix A. 



V. DISCUSSION A>ro CONCLUSION 

The research on which this is the first of a series of reports seeks to develop a means for measuring 
three task factors again:t common frames of reference. This is required to permit further research into 
methods for determining task training priorities. The method will be used to develop and validate a series of 
benchmark scales, the concept of which is discussed in this report. It has been established that supervisors 
can agree on the relative values of task lactors for tasks in an inventory drawn from many different 
specialties, and that these ratings can be used to develop the benchmark scales which define graduated levels 
for each of the task factors. The first set of scales have been drafted, and the broad .iiethod that wOl be 
used for their validation has been discussed. Results of this phase of the project will be the subject of ?. 
subsequent report, as will be the development of two further sets of benchmark scales. 
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APPENDIX A: BENCHMARK SCALES, DEFINITIONS AND EXPLANATIONS 

CONSEQUENCES OF INADEQUATE PERFORMANCE 
DEFINITION 

Consequences of inadequate performance is a measure of the seriousness of the probable consequences of inadequate 
performance of a task. It is measured in terms of possible injury or death, wasted supplies, damaged equipment, wasted 
man-hours of work, etc, 

BENCHMARK SCALE 

Level 1 — Least Serious Consequences of Inadequate Performance 
Deliver newspaper to local distribution points (Information Specialist AFSC 79150) 
Clean flight planning room (Command and Control Specialist AFSC 27450) 
Fold or count hospital linen (Medkral Service Specialist AFSC 90250) 

Level 2 

Tatto identification on Air Force working dogs (Veterinary Specialist AFSC 90850) 

Arrange, mark and display property for best sales results (Materiel Facilities Specialist AFSC 64750) 

Enter daily work assignments on time cards (Adrninistrstion Specialist AFSC 70250) 

Level 3 

Draw up work rosters for taxi operators or drivers on large AF base (Programs and Work Coniro* Specialist AFSC 55530) 
Compute selling price for processed meat and meat produce (Meatcutter AFSC 61 250) 
Compute quantity of earth to be removed or used for fill (Geodetic Surveyor AFSC 22250) 

Level 4 

Finish and polish gold alloy inlays, crowns or fixed partial dentures (Dental Laboratory Specialist AFSC 98250) 
Operate keypunch machine to keypunch data cards (Personnel Specialist AFSC 73250) 

Perform normal satellite photography sequence (Aerospace Control & Warning Systems Operator AFSC 27650) 
Level 5 

Reload computer after power failures or fluctuations (Communications Center Specialist AFSC 29150) 

Detect theft of rnoney or stock from commissaries or supply service outlets (Supply Services Specialist AFSC 61 1 50) 

Measure and record auditory acuity or hearing sensitivity (Aeromedical Specialist AFSC 90150) 

Level 6 

Quell disturbances involving military personnel (Security Specialist AFSC 81150) 
Prepare aircrew navigation kits (Air Operations Specialist AFSC 27150) 

Take and record pulses, temperatures and respirations (Medical Service Specialist AFSC 90250) 
Level? 

Fit cargo parachutes to airdrop cargo (Aircrew Life Support Specialist AFSC 9225C) 

Analyse radarscope photographs to identify tar jsts or evaluate target condition (Imagery Interpreter AFSC 20650) 
Sterilize surgical instruments or supplies (Operating Room Specialist AFSC 90252) 

Level 8 

Apply first aid at scene of accident or incident (Security Specialist AFSC 811 50) 

Render missile safe for maintenance cr verify missile safing (Missile Safety Specialist AFSC 241 50B) 

Alert direction finding (OF) station? when aircraft emergencies occur (Radio Operator AFSC 29353) 

Level 9 — Most Serious Consequences of Inadequate Performance 

Defend AF installations against attack by hostile forces or sabateurs (Security Specialist (Military Dog Qualified) AFSC 
81 150 A) 

Assist patient to maintain proper airway during surgery (Operating Room Specialist AFSC 90252) 
Rescue personnel from aircraft or aerospace vehicle (Fire Protection Specialist AFSC 57150) 

USE OF THE SCALE 

1. For each task in turn, think of the probable consequences of inadequate performance. Think in terms of possible injury 
or death, wasted supplies, damaged equipment, wasted man-hours or work, etc. 

2. Decide which set of three tasks in the above scale have about the same consequences of inadequate performance. 

2. The level indicated for this set of three tasks is your measure of the consequences of inadequate performance for the 
task urxier consideration. 
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APPENDIX A: (Continued) 

TASK DELAY TOLERANCE 

DEFINITION AND EXPLANATION 

The Task Delay Tolerance of a task is a measure of how much delay can be tolerated between the time the airman becomes 
aware the task is to be performed and the time he must corrvnence doing jt. Must he commence immediately, or does he 
have time to consult a manual, seek guidance, or even be taught how to do it? 



BENCHMARK SCALE 

Level 1 — Least Toleranoe of Delay 

Use artificial respiration to restore breathing of accident or fire victims (Fire Protection Specialist AFSC 57150) 
Issue scramble orders to fighter aircraft (Command and Control Specialist AFSC 27450) 

Assist during treatment of cardio-respiratory failure in operating room (Operating Room Specialist AFSC 90252) 
Level? 

Quell disturt>ances involving military personnel (Security Specialist AFSC 81 150) 

Identify tablets, capsules or lk)uids involved in poisoning cases (Pharmacy Specialist AFSC 90550) 

Operate safety console at missile control center during hazardous operations (Missile Safety Specialist AFSC 241 50B) 

Levels 

Inspect runway for foreign objects (Air Operations Specialist AFSC 271 50) 
Administer anaesthesia in dental surgery (Dental Specialist AFSC 98150) 
Adjust airtwrne radio receivers to obtain readable signals (Radio Operator AFSC 29353) 
Level 4 

Question suspects or witnesses (Security Specialist AFSC 81 1 50) 

Perform colony counts on bacteria to estimate type and level of infection (Medical Laboraton/ Specialist AFSC 90450) 

Maintain proper temperature of food storage areas (Cook AFSC 62250) 

Levels 

Identify military vehicles, installations or activities in visual photographs (Inteiligence Operations Specialist AFSC 20450) 
Proofread or con-ect teletype tape or page copies (Communeattons Center Specialist AFSC 29150) 
Prepare daily weather maps (Weather Forecaster Specialist AFSC 25330) 
Level 6 

Operate computer remote inquiry terminals (Computer Operator AFSC 51150) 

Purge or clear chemical lines in film developing machines (Still Photographic Laboratory Specialist AFSC 23354) 
Service and maintain dental high-speed drilling equipment (Dental Laboratory Specialist AFSC 98250) 
Level? 

Monitor workload reporting systems (Manpower Specialist AFSC 73330) 

Brief person nel on state or tocat motor traffic law: (Safety Specialist AFSC 241 50) 

"^Sr^JiK rosters for taxi operators or driven on large Air Force base (Programs and Work Control Specialist AFSC 
5o530i 

Level 8 

Write item identification descriptions and specifications for catalogues (Procurement Specialist AFSC 651 50) 

Interview or hire civiian personnel (Supply Services Specialist AFSC 61150) 

Prepare and anahfie work flow process charts (Management Engineering Specialist AFSC 73331 ) 

Level 9 - Moet Tolerant of Delay 

Review or select books or publications 1 r unit library (AdnHnistration Specialist AFSC 70250) 
Research and write feature stories in Air Fc.ce publications (lnformatk>n Specialist AFSC 79150) 
Clean teeth of animals (Veterinary Specialist AFSC 90850) 

USE OF THE SCALE 



* ^ ^'"'^ °' amount of time that could normally be allowed to elapse from the time the ainnan 

who has to do the task first becomes aware of the need to perform it and the time he must actually commence doing it 
Think in terms of time he could spend reading up on the task or seeking advice on how to do it, or iust thinking about 
how to do it. 

2. Decide which set of three tasks in the above scale have about the same amount of task delay tolerance. 

3, The level indicated for this set of three tasks is your measure of the task delay tolerance of the task under consideration. 
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APPENDIX A: (Continued) 

TASK DIFFICULTY 



DEFINITION 

Task Difficulty is a measure of the need for lengthy, systematic training before a new member of the appropriate 

Air Force specialty could perform the task adequately. It may be thought of as the difficulty involved in "picking up" the 

task on the iob >A/ithout any systematic training. 



BENCHfVIARK SCALE 

Lev0l 1 — Least Difficult Learn 

Collect food trays or serving units from patients in hospital vwards (Medical Service Specialist AFSC 90250) 
Clean display cases, furniture or fixture*, in commissary (Supply Services Specialist AFSC 61 150) 
Stamp time of receipt on incoming messages (Communications Center Specialist AFSC 29150) 

Lav«l 2 

Distribute administrative orders within unit (Administration Specialist AFSC 70250) 
Schedule health examinations for meat cutting personnel (Meatcutter AFSC 61250) 
Count property in warehouse bins or shelves (Inventory Management Specialist AFSC 64550) 

Levels 

Challange or identify unknown persons in vcinity of correctional facility (Corrections Specialist AFSC 81251) 
Deliver passenger manifests and allied documents to international border clearance authorities (Air Passenger Specialist 
AFSC 60550) 

Act as arnrwd escort for personnel transferring funds (Security Specialist AFSC 81550) 
Level 4 

Complete and submit Radiation Exposure Registration Form (AF Form 1520) (Environmental Health Specialist AFSC 
90750) 

Assemble shelter manager kits (Disaster Preparedness Specialist AFSC 24250) 
Maintain imprest or petty cash account (Procurement Specialist AFSC 65150) 

Levels 

Verify labels or instructions for handling radioactive substarx;es (Materiel Facilities Specialist AFSC 64750) 

L.spect buildings for termites or other wood destroyers (Entomology Specialist AFSC 56650) 

Prepare comparative productivity charts for work centers (Management Engineering Specialist AFSC 73331) 

Levels 

Control or extinguish structural fires (Fire Protection Specialist AFSC 57150) 

Calculate number or amount of each food item to be prepared for therapeutic diet (Diet Therapy Specialist AFSC 62251) 
Prepare injured personnel for evacuation by litter or ambulance (Aeromedical Specialist AFSC 90150) 

Level? 

Administer Intermittent Positive Pressure Breathing (IPPb) therapy (Medical Service Specialist AFSC 90250) 
Analyse computer stops for possible hardware malfunction (Supply Systems Specialist AFSC 64650) 
Determine position of aircraft by analysis of radarscope photographs after mission (Imagery Interpreter Specialist AFSC 
20650) 

Levels 

Operate safety con<to1o at missile control center during hazardous operations (Missile Safety Specialist AFSC 241508) 
Differemiate betv een actual targets and electrons countermeasures or decoys (Electronk: Warfare Countermeasures AFSC 
27631) 

Determine axis of attack for air-to-ground attack missions (Intelligence Operations Specialist AFSC 20450) 
Level 9 - Most Difficult to Learn 

Determine chemkrat composition of foreign made drugs (Pharmacy Specialist AFSC 90550) 

Perform deep roentgen therapy on tumor or cancer patients (Radiology Specialist AFSC 90350) 

Assist during treatment of cardioH'espiratory failure in operating room (Operating Room Specialist AFSC 90252) 

USE OF THE SCALE 

1. For each task in turn, think of its learning difficulty; think in terms of the need for lengthy, systematic training, or 
alternatively in terms of the difficulty in "pk:king up" the task on the iob without systematic training. 

2. Decide which of the three tasks in the above scale have about the same difficulty . 

3. The level indk:ated for this set of three tasks is your measure of the difficulty of the task under consideration. 
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